Papers with position encoding
Word Order Matters When You Increase Masking (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that word order information is not always useful for pre-trained models to solve downstream tasks. |
| Approach: | They examine the effect of removing position encodings on the pre-training objective itself to see whether models can reconstruct position information from co-occurrences. |
| Outcome: | The proposed model can reconstruct position information from co-occurrences without explicit encoding, but not on some tasks. |
Better Pre-Training by Reducing Representation Confusion (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-trained language models address information confusion in position encoding and model representations. |
| Approach: | They propose two techniques to improve pre-trained language models by decoupling directions and auxiliary regularizers. |
| Outcome: | The proposed techniques can improve pre-trained language models on GLUE benchmarks. |
Intent Contrastive Learning Based on Multi-view Augmentation for Sequential Recommendation (2025.coling-main)
Copied to clipboard
Bo Pei, Yingzheng Zhu, Guangjin Wang, Huajuan Duan, Wenya Wu, Fuyong Xu, Yizhao Zhu, Peiyu Liu, Ran Lu
| Challenge: | Existing work on intent-related models fails to capture long-term dependencies in user behavior and fails to effectively utilize item relevance. |
| Approach: | They propose a sequential recommendation framework that combine temporal variability with position encoding that has extrapolation properties to encode sequences, thereby expanding the model’s view of user behavior. |
| Outcome: | The proposed model improves on three real datasets by 0.8% to 14.7% compared to baselines. |
Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers (2026.findings-eacl)
Copied to clipboard
| Challenge: | Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. |
| Approach: | They propose a position encoding strategy that uses random float sampling to generalize to unseen lengths. |
| Outcome: | The proposed strategy can generalize to lengths unseen during training and in benchmarks. |
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation (2025.acl-long)
Copied to clipboard
Chuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong, Jiankai Sun, Jingyao Li, Minbin Huang, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li
| Challenge: | Extensive experiments demonstrate that treating attention as a feature map and applying convolution as . a processing method significantly enhances Transformer performance. |
| Approach: | They propose to use the convolution operator to mimic the processing methods in computer vision to treat attention as a feature map and apply it to neighboring attention scores across different heads. |
| Outcome: | The proposed model can be adapted to various attention-related models and achieves high performance. |
HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position (2024.acl-long)
Copied to clipboard
| Challenge: | Existing LLMs are constrained by their pre-trained context lengths, leading to performance issues . elucidating this limitation, we propose a training-free solution to the context length limitation in LLM applications . |
| Approach: | They propose a method that integrates hierarchical rotary position embedding into LLMs without extra training costs. |
| Outcome: | The proposed method improves performance on language modeling and long code completion tasks. |
DcLM: Output Length Control of Large Language Models via Dynamic Length Markers (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have limited awareness of output length, making it difficult to satisfy precise length requirements. |
| Approach: | They propose a model-agnostic approach that introduces dynamic length markers to guide length-controllable outputs. |
| Outcome: | The proposed method significantly reduces length deviation across multiple datasets. |
LLM as Effective Streaming Processor: Bridging Streaming-Batch Mismatches with Group Position Encoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for adapting LLMs to streaming rely on expensive re-encoding or limited scalability. |
| Approach: | They propose a group position encoding paradigm built on batch architectures to enhance consistency between streaming and batch modes. |
| Outcome: | The proposed method outperforms existing methods on cross-lingual and cross-modal tasks. |